BetaText: An Event Driven Text Processing and Text Analyzing System

BetaText can be described as an event driven pr(xluction system, in which (c~mbinations of) text events lead to certain actions, such as the printing of sentences that exhibit certain, say, syntactic phenomena. %~]e analysis mechanism used allows for arbitrarily complex parsing, but is particularly suitable for finite state i~arsing. A careful investigation of what is actually needed in linguistically relevant text processing resulted in a rather sn%all but carefully chosen set of "elementary actions" to be implemented. 1. Introdnction. The field of c~mputa'tior~][ linguistics seems, roughly speaking, to o~IiprJ.se two rather disjoint subfields, one J.n which the typical researcher predominantly occupies himself witJl problems such as "concordance generation", "backward sorting", "word frequencies" and so on, whereas the prototypic researd]er in tJ~e otJler field has things like "parsing strategies", "semantic representations" on top of his mind. qhis division into almost disjoint subfields is to be regretted, because we all are (or should be) students of one and the same thing language as it is. %~e responsibility for this sad state of affairs can probably be divided equal by the researchers in these two subfields: the "concordance makers" .~cause they seem so entirely ha~)py with rather unsophisticated cx)raputational tools de~eloped a].reac~ in the sixties (and which allow the researcher to look at words or word forms only, and their distribution), and the theoreticians ~yecause they seem so obsessed with the idea of developing their fantastic ir~dels of ]xln(}lage in greater and greater detail, a mode], that at a closer scrutiny is found to c~Dmprise a lexicon of, at best, a couple of hundred words, and cvavering, at best, a couple of hundred sentences or so. No wonder that the researchers in these two canlos thirJ< so little of each other. One way of closing the gap can be to develop niDre sophisticated tools for the investigation of actual texts; there is a need for die theoreticians to test to what extent their models actually cover actual language (and to get impulses from actual language), and there is a need for the "practicioners" to have simple tools for investigating snore complex st[llctures in texts than mere words and word :totals. BetaText is an attempt to provide tools for both those needs. 2. Text events and text oiyerations. BetaText is a system intended both for scientific investigations (or analyses) of texts, and text processing in a i~ore technical sense, such as reformattlng, washing spurious characters away, and so on. Due to the internal organisation of the system, even large texts can [se run at a reasonable cost (of. BroddaKarlsson ±98i). In this section we give some general definitions, and show their consequences for BetaTe xt. i~i elementary (text) event consists of the observation of one specified, concrete string in the text. The systera records sudl an observation through the introduction of a specific internal state (oz through a specific change of the internal state), the internal state being an internal variable that can take arbitrary, positive integral values. /Lrbitrarily chosen states (sets of states, in fact.) can be tied to specific activities (or pro cesses), and each time such a state is intro duced (i.e. the internal state becomes equal to that state) the corresponding process is aeti vated. Such states are called action states. A complex event (or just event, even elementary events can be. cor~lalex in the sense used here) is the c~3mbined result of a sequence of interconnected elementary events, possibly resulting in an action state. In BetaText all this is coi~pletely controlled by a set of prEx~uction rules (cf. Smullyan 196].) of the type~ (, ) -> ( , , , ) where is the string that is to be observed, a condition for applying the rule, viz. that the current inter]lal state belongs to this set; it is via such conditions that the chaining of several elementary events into one con~91ex event is achieved. is a string that is substituted for the observed string (the default is that the original string is retained), is a directive to Che system w'here (in the text) it shall continue the analysis; the default is immediately to